A New First-order Framework for Orthogonal Constrained Optimization Problems

In this paper, we consider a class of orthogonal constrained optimization problems, the feasible region of which is called the Stiefel manifold. Our new proposed framework combines a function value reduction step with a symmetrization step. Different from the existing approaches, the function value reduction is conducted in the Euclidean space instead of the Stiefel manifold or its tangent space. We construct two types of algorithms based on this new framework. The first type is gradient reduction based algorithms which consists of gradient reflection (GR) and gradient projection (GP) two implementations. The other one adopts a column-wise block coordinate descent (CBCD) scheme with a novel idea for solving the corresponding CBCD subproblem inexactly. Theoretically, we can prove that both GR/GP with a fixed stepsize and CBCD belong to our framework, and any clustering point of the iterates generated by the proposed framework is a first-order stationary point. Preliminary experiments illustrate that our new framework is of great potential.